Skip to content

docs: when Browser checks and Playwright Check Suites keep traces and videos - #813

Merged
hlenke merged 3 commits into
mainfrom
docs/videos-traces-when-recorded
Nov 10, 2026
Merged

hlenke merged 3 commits into
mainfrom
docs/videos-traces-when-recorded

Conversation

@hlenke

@hlenke hlenke commented Nov 8, 2026 •

Copy link
Copy Markdown
Member

What changed and why

People often ask why a Browser check result has no trace or video, how to get them for passing runs, and how to turn video off. The facts were spread across the Playwright support page, and the Playwright Check Suite page described an old trace default.

  • Browser checks → Troubleshooting: new "Missing traces or videos" entry. It covers which runs keep artifacts, that the trace and video Playwright options are ignored (use a Playwright Check Suite to control them), that pages from a manually created browser.newContext() have no video, and that the test timeout should stay below the 240-second run limit.
  • Playwright Check Suites → Configuration: use.trace moves from "enforced: on for every run" to the defaults table as on-first-retry. A short paragraph explains that Checkly always records a trace but keeps the file only when your use.trace setting would have produced one, and what the defaults mean for passing and failing tests.
  • Browser checks → Timeouts: the run-limit heading now quotes the message users see, Timed out waiting 240s for the test suite to run, instead of "maximum run time of 240000 ms". "No assets will be available" becomes: the test still running at the limit has no trace or video.
  • Browser checks → Playwright support: "a video for each page navigation" becomes "a video for each page". Playwright records one video per page, not per navigation.

How each claim was verified

Browser check and Playwright Check Suite runners can't run in an isolated local devenv. Each claim was tested with real Playwright 1.58.2 runs, using the config that Checkly's own code generates on monorepo main (e086a77755):

  • Browser checks: getPlaywrightTestConfig with the runner's CLI flags (--global-timeout=240000, a kill at 250 s).
  • Playwright Check Suites: applyChecklyDefaults.
Claim How it was verified
Scheduled Browser check runs keep the trace and video only on failure Generated scheduled config is retain-on-failure. Passing run: no artifacts. Failing run: video.webm and trace.zip
Editor runs keep them for passing runs too Generated ad-hoc config is on. Passing run: video.webm and trace.zip
Browser checks ignore use.trace / use.video Passing use: { video: 'off', trace: 'off' } into the generator still yields retain-on-failure / on (same in the runner-ng copy, browser-runner.ts)
One video per page in the test's context Test opening a second page with context.newPage(): video.webm and video-1.webm
browser.newContext() pages have no video Failing test using a manual context: trace only, no video (Playwright videos guide)
A 210 s test timeout keeps trace and video Failing at the test timeout after 215 s: report attaches video, trace and error context
A test still running at the 240 s limit has none Report: "Timed out waiting 240s for the test suite to run", test not run, no attachments. The runner only collects per-test trace.zip files and reported videos (browser-assets.ts)
Suite default use.trace is on-first-retry; recording forced to on applyChecklyDefaults output (trace: 'on', retention on-first-retry). Unit tests: wrapperHelpers.test.ts (114 pass)
The trace is kept per use.trace Reporter trace-extraction tests (37 pass, shouldKeepTrace). use: { trace: 'off' } gives retention off, trace: 'on' gives on
With defaults, a passing test keeps no trace or video; a failing test keeps them only for its first retry; with 0 retries, neither Real run with the generated config. Pass: trace recorded, no video, and retention drops it. Fail: video only on retry 1, trace kept only for retry 1. --retries=0: no video, trace dropped
use.video: 'off' turns video off Generated config keeps the user's off
Runs recorded with npx checkly test --record and npx checkly trigger --record keep the trace and video of a passing Browser check Real runs on a production test account (CLI 10.1.0, runtime 2026.04): a passing check run with each command; GET /v1/test-sessions/{id}/results/{resultId}/assets lists video.webm and the trace zip for both
The run-limit error is Timed out waiting 240s for the test suite to run; the test still running has no trace or video Real run on a production test account: two tests of 150 s each. The result's errors are Timed out waiting 240s for the test suite to run and … for the teardown for test suite to run; assets hold only the first test's video and trace
Videos, traces and screenshots need assets:read (already in the permissions reference, not changed here); every built-in role has it Local stack, seeded custom-role members: a member without assets:read (with checks:read) gets 403 from the web app's asset routes (screenshot signed URL, navigation trace read) and from the public API (/next/assets/… screenshot link and trace read, /v1/check-results/{checkId}/{checkResultId}/assets); the owner gets 200 from the asset reads. The same member moved to Read Only, Read & Run, Read & Write and Admin gets 200 from each asset read (the /v1/check-results manifest passes the permission check and answers 404, as the seeded asset has no stored result). Built-in role lists on monorepo main: all five roles hold assets:read

Not tested / left out

  • No full check run through the API, results pipeline, storage and result page on a local stack. The browser runners only run in the default local environment, so most claims above are tested at the runtime level. The CLI --record and run-limit claims were tested with real runs on a production test account.
  • Left out pending a product fix: the Playwright Check Suite screenshot default.
  • Who can see videos and traces: no change here. The permissions reference already lists assets:read (View result assets) since docs: add a complete permissions reference #581; the test is in the table above.
  • Overlap: docs: sentence case for headings and titles #774 (draft, sentence case) touches the same three files. This PR only edits other lines, apart from adding a row to the defaults table.

🤖 Generated with Claude Code

…aces and videos

Add a troubleshooting entry for missing traces and videos in Browser checks,
correct the Playwright Check Suite trace default (on-first-retry, retention
follows use.trace), and say Browser checks record one video per page.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@mintlify

mintlify Bot commented Nov 8, 2026 •

Copy link
Copy Markdown
Contributor

Preview deployment for your docs. Learn more about Mintlify Previews.

Project Status Preview Updated
checkly-422f444a 🟢 Ready View Preview Nov 10, 2026, 1:07 AM

💡 Tip: Enable Automations to automatically generate PRs for you.

…eout message

Tested on a production account: a passing Browser check run with
checkly test --record and checkly trigger --record keeps video and trace.
A run past 240 s fails with 'Timed out waiting 240s for the test suite to run'.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

@hlenke hlenke left a comment

Copy link
Copy Markdown
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Needs changes · The trace and video claims check out against the runner source. Two things to fix first: the Playwright Check Suite construct page still says traces are kept on every run, and the renamed timeout heading drops an error message some readers still see.

Sibling page that now disagrees (not in the diff). constructs/playwright-check.mdx:157 lists the defaults as workers: 4, retries: 2, use.video: "on-first-retry" and use.screenshot: "on-first-retry", then says "Checkly also records traces on every run". After this PR, detect/synthetic-monitoring/playwright-checks/configuration.mdx lists use.trace: on-first-retry as an overridable default and says the trace is kept only when use.trace would have produced one. A reader of the construct page will expect a trace for every run. Suggested replacement for that line:

Checkly applies these Playwright defaults when you do not set them yourself: workers: 4, retries: 2, use.trace: "on-first-retry", use.video: "on-first-retry", and use.screenshot: "on-first-retry". Checkly records a trace on every run and keeps it according to your use.trace setting. It also uses the Checkly reporter and appends a Checkly user-agent suffix. Checkly supports Playwright's globalTimeout; the timer starts when npx playwright test begins.

The two guides that already describe this (guides/monitoring-ecommerce-apps-using-playwright.mdx:179, guides/playwright-testing-to-monitoring.mdx:120) agree with this PR.

Overlap. #818 also edits detect/synthetic-monitoring/browser-checks/troubleshooting.mdx. The hunks don't overlap: this PR adds an accordion after Performance and Timeout Issues; #818 edits the Timing and Race Conditions list and appends a new section at the end. Either order merges cleanly. I'd land this one first because it's smaller and #818's new intro links to the same trace docs. Once both land, the page says twice that pages from browser.newContext() have no video (item 3 here, and #818's "Tests share a page or context from beforeAll"). That's flagged on #818.

CI: all checks pass. Mergeable: clean. main has not changed the touched files since the merge base.

Inline: 1 important.

What I checked
  • Read the full diff from the merge base e0e0277f and the touched pages at the PR head: browser-checks/troubleshooting.mdx, timeouts.mdx, playwright-support.mdx, and playwright-checks/configuration.mdx.
  • Local app unavailable (BUSY: another Tilt on the ports), so I verified against source instead of end-to-end runs. The PR description reports real Playwright runs and production test-account runs. I didn't repeat those.
  • Browser check trace/video modes: monorepo main packages/check-runtimes-common/src/browser/getPlaywrightTestConfig.js and packages/playwright-check-runner/src/browser/browser-runner.ts. Both set trace and video to on for ad-hoc runs and retain-on-failure otherwise, after the user's use, so the user's trace/video are overridden. Ad-hoc includes runs with a test session and test result, which is what --record creates. This matches playwright-support.mdx (trace/video marked ❌ in the Use tab).
  • Playwright Check Suite defaults: packages/playwright-check-runner/src/lib/wrapperHelpers.ts. The default use.trace is on-first-retry; recording is forced to on; the requested mode is passed to the reporter as retention. Video and screenshot defaults are on-first-retry, and retries defaults to 2. These match the new table row and paragraph.
  • Run-limit message: for @playwright/test scripts the runner passes the execution timeout as Playwright's global timeout, which yields Timed out waiting 240s for the test suite to run. Scripts that don't use @playwright/test still fail with Your check run has reached the maximum run time of … ms. The run timeout has no Playwright global timeout in front of it there (browser-runner.ts, run-timeout handling; legacy apps/runners/functions/src/browser-check-common/src/AbstractBrowserRuntime.js).
  • Playwright videos guide (https://playwright.dev/docs/videos): manually created contexts are handled separately from the test fixtures' video option.
  • Siblings: grepped all .mdx for trace retention claims (records traces on every run, use.trace, traces are only).
  • scripts/pr_impact.py: no findings. Anchors #viewing-trace-files and #checkly-applied-playwright-settings exist at the PR head.
  • Not verified: the video-per-page and browser.newContext() claims beyond the Playwright docs; I relied on the author's runs.

@hlenke
hlenke enabled auto-merge (squash) November 10, 2026 01:08
@hlenke
hlenke merged commit cdb4655 into main Nov 10, 2026
7 checks passed
@hlenke
hlenke deleted the docs/videos-traces-when-recorded branch November 10, 2026 01:09

This branch was successfully deployed

1 active deployment
staging — 6c06908e Deployed Nov 10, 2026 by mintlify[bot]
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant